Tracking and Recognising Hand Gestures using Statistical Shape Models

Abstract Hand gesture recognition from video images is of considerable interest as a means of providing simple and intuitive man-machine interfaces. Possible applications range from replacing the mouse as a pointing device to virtual reality and communication with the deaf. We describe an approach to tracking a hand in an image sequence and recognising, in each video frame, which of five gestures it has adopted. A statistically based Point Distribution Model (PDM) is used to provide a compact parameterised description of the shape of the hand for any of the gestures or the transitions between them. The values of the resulting shape parameters are used in a statistical classifier to identify gestures. The model can be used as a deformable template to track a hand through a video sequence but this proves unreliable. We describe how a set of models, one for each of the five gestures, can be used for tracking with the appropriate model selected automatically. We shown that this results in reliable tracking and gesture recognition for two 'unseen' video sequences in which all the gestures are used.